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Abstract 

Computer  vision  software  is  complex,  involving  many  tens  of  thousands  of  lines  of  code. 
Coding  mistakes  are  not  uncommon.  When  a  vision  algorithm  is  run  on  controlled  data 
which  meet  all  the  algorithm  assumptions,  the  results  are  often  statistically  predictable. 
This  renders  it  possible  to  statistically  validate  the  algorithm  and  its  associated  theo¬ 
retical  derivations.  In  this  paper  we  review  the  general  theory  of  some  relevant  kinds 
of  statistical  tests  and  then  illustrate  the  experimental  methodology  of  statistical  algo¬ 
rithm  validation  to  validate  a  program  that  estimates  parameters  of  buildings  in  aerial 
photographs.  This  program  estimates  the  3D  positions  of  building  vertices  based  on 
input  data  obtained  from  multi-image  photogrammetric  resection  calculations  and  3D 
geometric  information  relating  some  of  the  points,  lines  and  planes  of  the  building  to 
each  other. 
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Abstract 

Compnter  vision  software  is  complex,  involving  many  tens  of  thonsands  of  lines  of  code. 
Coding  mistakes  are  not  nncommon.  When  a  vision  algorithm  is  rnn  on  controlled  data 
which  meet  all  the  algorithm  assnmptions,  the  resnlts  are  often  statistically  predictable. 
This  renders  it  possible  to  statistically  validate  the  algorithm  and  its  associated  theo¬ 
retical  derivations.  In  this  paper  we  review  the  general  theory  of  some  relevant  kinds 
of  statistical  tests  and  then  illnstrate  the  experimental  methodology  of  statistical  algo¬ 
rithm  validation  to  validate  a  program  that  estimates  parameters  of  bnildings  in  aerial 
photographs.  This  program  estimates  the  3D  positions  of  bnilding  vertices  based  on 
inpnt  data  obtained  from  mnlti-image  photogrammetric  resection  calcnlations  and  3D 
geometric  information  relating  some  of  the  points,  lines  and  planes  of  the  bnilding  to 
each  other. 
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1  Introduction 


Many  computer  vision  problems  can  be  posed  as  either  parameter  estimation  problems 
(for  example,  estimate  the  pose  of  an  object)  or  hypothesis  testing  problems  (for  example, 
which  of  N  objects  in  a  database  occurs  in  a  given  image.)  Since  the  input  data  (such  as 
images  or  feature  points)  is  noisy,  the  estimates  produced  by  computer  vision  algorithms 
are  noisy.  In  other  words,  there  is  an  inherent  uncertainty  associated  with  the  results 
produced  by  any  computer  vision  algorithm.  These  uncertainties  are  best  expressed  in 
terms  of  statistical  distributions  and  their  means  and  covariances.  Details  of  the  theory 
and  application  of  covariance  propagation  can  be  found  in  [5,  8],  and  in  the  references 
cited  in  [8]. 

Usually,  implementations  of  vision  algorithms  involve  thousands  of  lines  of  code.  Fur¬ 
thermore,  the  algorithms  are  often  based  on  many  approximations  and  mathematical 
calculations.  One  way  to  check  whether  the  implementation  and  the  theoretical  calcula¬ 
tions  are  correct  is  by  providing  the  algorithm  with  input  data  with  known  (controlled) 
statistical  characteristics  (which  is  possible  since  the  input  data  can  be  artihcially  gener¬ 
ated)  and  then  checking  if  the  distribution  of  the  output  is  in  agreement  with  what  was 
predicted  by  theoretical  calculations. 

Since  many  of  the  estimation  problems  in  computer  vision  are  multidimensional, 
testing  whether  the  means  and  covariances  of  the  empirical  distribution  and  predicted 
distribution  are  the  same  is  easier  than  testing  whether  the  shapes  of  the  two  distributions 
are  the  same.  In  this  paper,  we  summarize  statistical  tests  for  the  case  in  which  the 
estimates  can  be  assumed  to  be  multivariate  Gaussian.  We  also  describe  the  interfaces 
to  software  that  we  have  implemented  for  conducting  these  tests.  Although  software 
libraries  and  environments  (e.g.  Spins,  numerical  recipes)  are  available  for  conducting 
such  tests  for  one-dimensional  samples,  we  are  unaware  of  similar  software  libraries  for 
the  multivariate  case.  In  fact,  most  statistics  books  do  not  give  all  hve  of  the  tests  we 
give  (e.g.,  Koch  [12]  does  not  address  the  hfth  testing  problem).  The  hypothesis  testing 
theory  and  software  are  described  in  [fO],  and  the  software  can  be  obtained  at  no  cost 
from  the  Statlib  software  repository  or  from  kanungo@cfar.umd.edu.  A  description  of 
how  the  software  and  the  theory  have  been  tested  using  statistical  techniques  is  also 
included.  A  preliminary  version  of  this  work  was  presented  at  the  1996  DARPA  Image 
Understanding  Workshop  [16]. 

2  Kinds  of  Statistical  Hypotheses 

Let  Xi,  X2,  ■  ■  . ,  Xn  be  a  sample  from  a  multivariate  Gaussian  distribution  with  population 
mean  //  and  population  covariance  S.  That  is,  Xi  G  and  Xi  ~  fV(//,  S),  where  p  is  the 
dimension  of  the  vectors  Xi. 

We  can  formulate  various  hypotheses  about  the  population  mean  and  covariance, 
depending  on  what  is  known  and  what  is  unknown.  The  data  Xi  can  then  be  used  to 
test  whether  a  hypothesis  is  false.  Notice  that  each  population  parameter  (here  we  have 
two,  p  and  S)  can  be  either  tested,  or  unknown  and  untested,  or  known.  If  a  parameter 
is  being  tested,  a  claim  about  its  value  is  being  made.  If  a  parameter  is  unknown  and 
untested,  no  claim  is  being  made  about  its  value;  its  value  is  not  known  and  therefore  we 
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cannot  nse  it  in  any  compntation.  If  the  valne  of  a  parameter  is  assnmed  to  be  known, 
then  we  assnme  that  its  valne  is  known  withont  error  and  cannot  be  qnestioned  or  tested, 
jnst  as  the  normality  assnmption  is  not  qnestioned.  Fnrthermore,  when  a  parameter  valne 
is  known,  that  valne  can  be  nsed  in  compntations  of  test  statistics  for  other  parameters. 

In  general,  if  a  distribntion  has  q  parameters,  there  can  be  3^  —  2^  tests.  The  reasoning 
is  as  follows.  Since  each  parameter  can  be  either  tested,  or  nnknown  and  nntested,  or 
known,  the  nnmber  of  possibilities  is  3^.  Bnt  of  these,  the  nnmber  of  combinations  in 
which  none  of  the  parameters  are  tested  (that  is,  they  are  either  known,  or  nnknown 
and  nntested  and  so  do  not  represent  a  test)  is  2^.  Thns  the  total  nnmber  of  distinct 
hypotheses  that  can  be  made  abont  a  sample  from  a  ^'-parameter  distribntion  is  3^  —  2^. 

In  the  case  when  the  data  come  from  a  mnltivariate  normal  distribntion,  S), 

we  have  q  =  2  and  thns  can  have  3^  —  2^  =  5  possible  hypotheses.  The  hve  tests  are  as 
follows: 

Hi  :  //  =  //o  (S  =  Si  known.)  In  this  test,  the  qnestion  is  whether  or  not  the  sample 
is  from  a  Ganssian  popnlation  whose  mean  is  //q-  The  popnlation  covariance  S  is 
assnmed  to  be  known  and  eqnal  to  Si. 

H2  :  fJ,  =  f-io  (S  unknown,  untested.)  In  this  test,  the  qnestion  is  whether  or  not  the 
sample  is  from  a  Ganssian  popnlation  whose  mean  is  //q-  No  statement  is  made 
regarding  the  popnlation  covariance  S. 

H3  :  S  =  So  (//  =  //i  known.)  In  this  test,  the  qnestion  is  whether  or  not  the  sample 
is  from  a  Ganssian  popnlation  whose  covariance  is  Sq.  The  popnlation  mean  //  is 
assnmed  to  be  known  and  eqnal  to  //i. 

Hi  :  S  =  So  (//  unknown,  untested.)  In  this  test,  the  qnestion  is  whether  or  not  the 
sample  is  from  a  Ganssian  popnlation  whose  covariance  is  So.  No  statement  is  made 
regarding  the  mean  //. 

H3  \  //  =  //o  S  =  So  In  this  test,  the  qnestion  is  whether  or  not  the  sample  is  from  a 
Ganssian  popnlation  whose  mean  is  //o  and  whose  covariance  is  So.  This  is  the 
principal  test  we  nse  for  software  validation. 

3  Definitions 

In  this  section  we  briefly  dehne  the  terms  nsed  in  the  rest  of  the  paper.  A  reader 
who  is  familiar  with  statistics  can  skip  this  section.  For  a  Incid  explanation  of  the 
basic  nnivariate  concepts  see  [4].  A  slightly  more  rigorons  treatment  of  nnivariate  and 
mnltivariate  tests  is  given  in  [2].  Mnltivariate  tests  are  treated  in  great  detail  in  [12]. 
The  most  anthoritative  reference  on  mnltivariate  statistics  is  [fj.  Althongh  this  book  has 
most  of  the  resnlts  given  here,  it  is  not  easy  to  read,  and  the  resnlts  are  not  easy  to  hnd 
in  it. 

A  statistic  of  the  data  Xi, .  .  .  ,  is  any  fnnction  of  the  data.  For  example,  the  sample 
mean  x  is  a  statistic,  and  so  is  the  sample  covariance  matrix  S.  The  statistic  need  not 
be  one-dimensional;  (x,  Sy  together  are  another  statistic  of  the  same  data.  A  sufficient 
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statistic  is  a  statistic  that  contains  all  the  information  abont  the  data;  any  inference 
regarding  the  nnderlying  popnlation  can  be  made  nsing  jnst  the  snfficient  statistic,  and 
the  individnal  data  points  do  not  add  any  more  information  to  the  inference  process. 
For  example,  the  vector  of  original  data  (xi, .  .  . ,  XnY  is  a  snfficient  statistic;  it  contains 
all  the  information  abont  the  data.  Another  snfficient  statistic  is  (x,  Sy.  A  snfficient 
statistic  is  not  nniqne.  A  minimal  sufficient  statistic  is  a  snfficient  statistic  that  has  a 
smallest  number  of  entries.  For  example,  for  Ganssian  data,  [x,  S)  is  a  minimal  snfficient 
statistic. 

A  hypothesis  is  any  statement  abont  a  popnlation  parameter  that  is  either  trne  or 
false.  The  null  hypothesis  Hq  and  the  alternate  hypothesis  Ha  are  the  two  complementary 
hypotheses  in  a  statistical  hypotheses  testing  problem. 

A  test  statistic  is  a  statistic  of  the  data  that  is  nsed  for  testing  a  hypothesis.  The  null 
distribution  is  the  distribntion  of  the  test  statistic  when  the  nnll  hypothesis  is  trne.  The 
alternate  distribution  is  the  distribntion  of  the  test  statistic  when  the  alternate  hypothesis 
is  trne. 

There  are  two  types  of  errors  —  mis-detection  and  false  alarm.  If  the  nnll  hypothesis 
is  trne  bnt  the  test  procednre  decides  it  is  false,  this  is  called  a  misdetection.  When  the 
alternate  hypothesis  is  trne  bnt  the  test  procednre  accepts  the  nnll  hypothesis,  this  is 
called  a  false  alarm.  The  misdetection  probability  of  a  test  procednre  is  nsnally  hxed  by 
the  nser;  this  is  referred  to  as  the  significance  level,  a,  of  the  test.  A  typical  valne  for  a 
is  0.05. 

The  power  function  of  a  hypothesis  test  is  a  fnnction  of  the  popnlation  parameter 
6]  its  valne,  /3(0),  is  eqnal  to  1  minns  the  probability  of  false  alarm.  Ideally,  the  power 
fnnction  shonld  be  0  for  all  6  snch  that  the  nnll  hypothesis  is  trne,  and  1  for  all  6  snch  that 
the  alternate  hypothesis  is  trne.  For  most  realistic  testing  problems  one  cannot  create 
a  test  procednre  with  snch  an  ideal  power  fnnction.  Power  fnnctions  are  very  nsefnl  for 
evalnating  hypothesis  testing  procednres.  For  an  example  in  which  they  are  nsed  for 
compnter  vision  problems  see  [If].  A  uniformly  most  powerful  test  is  a  test  procednre 
whose  power  fnnction  is  higher  than  those  of  all  other  tests. 

There  are  many  methods  of  creating  tests  and  corresponding  test  statistics.  The 
test  statistics  given  in  this  paper  were  derived  by  maximizing  the  likelihood  ratio.  The 
derivation  of  these  statistics  can  be  fonnd  in  the  cited  literatnre. 

4  Test  statistics 

In  this  section  we  snmmarize  varions  test  statistics  and  their  distribntions  nnder  the  trne 
nnll  hypothesis  For  details  see  Anderson  [1]  and  Kannngo  and  Haralick  [10].  These  tests 
will  be  nsed  later  in  onr  discnssion.  We  nse  the  following  dehnitions  of  x  and  S  : 

1  ” 

X  =  -  V  x* 
n 


and 

1  ” 

S  =  - -  Y^{xi  -  .x){.Xi  -  .xf, 

n  —  1  ^ 
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where  we  have  assumed  that  the  data  vectors  Xi  are  p-dimensional  and  the  sample  size 
is  n. 

4.1  Test  1:  ji  =  jio  with  known  S  =  Si 

Test  statistic: 

r  =  n(x  - - //o).  (1) 

Its  distribution  under  the  null  hypothesis  is  Chi-squared: 

T-xl- 

The  alternate  hypothesis  is  Ha  '■  fJ-  ^  fJ-o]  the  distribution  of  the  test  statistic  under  the 
alternate  hypothesis  is  non-central  Chi-squared: 

rri  2 

Xp^d 

where  d  =  n{ji  —  —  //q)  is  the  non-centrality  parameter. 

4.2  Test  2:  ji  =  jio  with  unknown  S 

Hotelling’s  Test  statistic: 

T  =  (x  -  -  //o)-  (2) 

Its  distribution  under  the  null  hypothesis  is  F: 

T  ^  F 

J-  p,n  —  p  • 

The  alternate  hypothesis  is  Ha  '■  fJ-  ^  fJ-o]  the  distribution  of  the  test  statistic  under  the 
alternate  hypothesis  is  non-central  F: 

T  ^  P  , 

p,n—p,d 

where  d  =  n(fi  —  —  //q)  is  the  non-centrality  parameter. 

4.3  Test  3:  S  =  So  with  known  //  = 

Let 

n 

^  “  Fi){xi  -  liif  =  (n  -  l)^  +  (x  -  Iii){x  -  jiif  . 

and 

A  =  (e/nf”/2|C'S-^r/2exp(-fr(C'So-^)/2)  . 

The  test  statistic  is 

T  =  —2  log  A.  (3) 

Its  distribution  under  the  null  hypothesis  is  Chi-squared: 

^  X-p(-p+l)j2- 

The  alternate  hypothesis  is  Ha  '■  S  ^  Sq;  the  distribution  of  the  test  statistic  under  the 
alternate  hypothesis  is  unknown. 
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4.4  Test  4:  S  =  So  with  unknown  // 

Let  B  =  {n  —  1)5',  and  A  =  (e/(n  —  exp(— tr(_BSo  ^)/2)  The  test 

statistic  is 

T  =  —2  log  A.  (4) 

Its  distribution  under  the  true  hypothesis  is  Chi-squared: 

^  ~  Sp(p-|-l)/2- 

The  alternate  hypothesis  is  Ha  '■  S  ^  Sq;  the  distribution  of  the  test  statistic  under  the 
alternate  hypothesis  is  unknown. 

4.5  Test  5:  S  =  So  and  ji  =  jio 

Dehne  B  =  [n  —  \)S  and 

A  =  (e/n)P”/2|5So-i|”/2 

exp  (|-[fr(i?So  +  n{x  -  //o)*So  -  ho)]/2) 

The  test  statistic  is 

T  =  —2  log  A  (5) 

Its  distribution  under  the  null  hypothesis  is  Chi-squared: 

T  ~  Xp(pAi)/2+p 

The  alternate  hypothesis  is  Ha  '■  S  ^  So,  and  //  ^  fj,o]  the  distribution  of  the  test  statistic 
under  the  alternate  hypothesis  is  unknown. 

5  Validating  theory  and  software 

To  validate  computer  vision  software  two  checks  have  to  be  performed.  The  hrst  check  is 
that  the  theory  is  correct:  the  theoretically  derived  null  distributions  of  the  test  statistics 
are  actually  correct.  The  second  check  is  that  the  software  is  correct:  the  implementa¬ 
tion  is  exactly  what  the  theory  dictates.  Both  of  these  checks  can  be  done  by  computing 
the  empirical  distributions  and  comparing  them  with  the  theoretically  derived  distribu¬ 
tions.  In  the  next  subsection  we  describe  how  we  empirically  compute  the  null  distri¬ 
butions  of  the  hve  test  statistics,  and  in  the  following  section  we  describe  how  we  use 
the  Kolmogorov- Smirnov  test  to  check  if  the  empirical  distribution  and  the  theoretically- 
derived  distributions  are  the  same. 

5.1  Empirical  null  distributions 

In  order  to  generate  the  empirical  null  distributions  we  proceed  as  follows. 

1.  Choose  some  values  for  the  multivariate  Gaussian  population  parameters  p, //  and 

S. 
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2.  Generate  n  samples  from  the  population. 

3.  Compute  the  value  of  the  statistic  T  for  the  test  that  is  being  verihed. 

4.  Repeat  steps  2  and  3  M  times  to  get  T*,  i  =  f , .  .  .  ,  M. 

5.  The  empirical  distribution  T  can  be  computed  by  computing  the  histogram  of  T^. 

5.2  Kolmogorov-Smirnov  test 

The  Kolmogorov-Smirnov  (KS)  test  tests  whether  two  distributions  are  alike.  The  KS 
test  uses  the  fact  that  the  maximum  absolute  difference  between  the  empirical  cumulative 
distribution  (the  KS  test  statistic)  and  the  theoretical  cumulative  distribution  has  a 
known  distribution  (the  null  distribution).  For  a  more  detailed  discussion  of  the  KS  test 
see  [f7]. 

6  Application:  3D  Parameter  Estimation 

We  applied  our  hypothesis  testing  methodology  to  validate  the  3D  parameter  estima¬ 
tion  software  that  was  used  for  constructing  the  ground  truth  model  from  the  RADIUS 
model  board  data  set  [18].  In  this  section  we  describe  the  problem  and  the  optimization 
approach. 

6.1  Site  Model  Construction 

The  task  is  to  construct  3D  object  models  from  detected  2D  image  features  and  the 
known  geometric  constraints  on  the  observed  perspective  projections  of  the  3D  objects. 
The  data  set  [18]  consists  of  78  images  of  the  two  RADIUS  model  boards  and  the  3D 
coordinates  of  some  building  vertices.  Since  the  purpose  was  to  establish  ground  truth  for 
automatic  site  model  construction  algorithms,  the  corresponding  points  of  the  building 
vertices  that  were  observable  on  the  images  were  identihed  and  located  manually.  Also, 
the  3D  positions  of  a  few  of  the  building  vertices  are  known.  Simultaneous  estimation 
of  the  interior  parameters  and  exterior  orientation  parameters  of  the  cameras  was  done 
by  setting  up  and  solving  a  very  large  photogrammetric  resection  problem.  Then,  using 
these  camera  parameters,  multi-image  triangulation  was  performed.  This  yielded  noisy 
estimates  for  the  building  vertices;  these  estimates  were  the  input  to  the  site  model 
construction  software  whose  testing  we  now  describe. 

The  geometric  constraint  procedure  takes  the  photogrammetrically  estimated  3D 
point  positions  and  their  covariance  matrices  as  observations,  ft  uses  the  partial  models 
of  the  buildings  to  generate  constraints  on  the  building  parameters.  To  estimate  the 
optimal  3D  parameters  that  satisfy  the  relations  in  the  partial  models,  a  constrained  op¬ 
timization  model  is  solved.  By  error  propagation  we  derive  the  covariance  matrix  of  the 
estimated  building  vertices,  which  are  now  guaranteed  to  satisfy  the  given  constraints. 
This  is  discussed  in  greater  detail  in  [15]. 
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6.2  Constrained  Optimization 

Partial  building  models  are  used  to  constrain  the  3-D  parameters  to  be  estimated.  Par¬ 
tial  models  represent  geometric  constraints  between  building  entities:  lines,  planes  and 
vertices.  The  specihc  relations  we  used  to  constrain  the  parameters  were:  point-on- 
plane,  point-on-line,  plane-angle-plane,  line-angle-line,  and  plane-angle-line.  For  a  more 
detailed  description  see  [13]. 

The  observed  3D  points  and  the  associated  covariance  matrix  S  are  obtained  by 
triangulation.  The  perturbation  model  that  we  used  for  the  observations  was  zero-mean 
Gaussian  noise  with  unknown  covariance.  Having  the  partial  object  model  and  the 
perturbation  model,  we  can  dehne  the  estimation  problem.  Let  0  G  ]R™  denote  the 
parameters,  X'  G  ]R™  the  observations,  and  p{X'  \  0)  the  likelihood  function.  In  the 
building  estimation  problem,  the  parameters  are  the  coordinates  of  the  points,  the  normal 
vectors  and  distance  constants  of  the  planes,  and  the  direction  cosines  and  reference  points 
of  the  lines. 

Assuming  that  the  optimality  criterion  is  the  maximum  posterior  probability,  a  Bayesian 
approach  can  be  used  to  transform  the  problem  into  a  maximum  likelihood  problem  with 
constraints.  Let  the  constraints  be  denoted  by  0  G  Cq  C  Jf™.  This  problem  can  be 
expressed  as  a  constrained  optimization  problem. 

min{— p(X'  I  0)  I  0  e  Ce} 

The  problem  can  be  reformulated  by  taking  the  logarithm  of  the  probability  function. 
Under  the  assumption  of  Gaussian  noise,  we  obtain  a  least  squares  model.  The  objective 
function  is  the  sum  of  squared  errors  between  the  estimated  point  positions  and  the 
observed  points. 

mm  {/(0|  :=  (V  -  XfE-\X'  -  A')}  (6) 

subject  to  0  G  C@ 

where  X  denotes  the  unknown  3D  points,  and  the  feasible  set  Ce  is  determined  by  the 
partial  model  and  the  unit  length  constraint  on  the  directional  vectors. 

If  the  noise  affecting  different  3D  points  is  independent,  the  objective  function  can  be 
rewritten  as 

/(©)  =  -  X,) 

*=i 

where  is  the  covariance  matrix  of  the  iih  point  and  K  is  the  number  of  observed 
points. 

The  constraints  can  be  incorporated  into  the  optimization  problem  as  follows  (see 

[13]): 

mm  {/(0|  :=  (X'  -  XfE-^X'  -  A)}  (7) 

subject  to  hi{Q)  =  0,  i  =  1,  ...,r 

In  the  above,  the  equation  hi{Q)  =  0  represents  all  the  constraints  derived  from  the 
partial  models.  The  numerical  solution  to  this  optimization  problem  can  be  achieved 
by  various  methods  such  as:  the  reduced  gradient  method  [7],  sequential  quadratic  pro¬ 
gramming  [6],  or  the  augmented  Lagrangian  method  [3]. 
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7  Error  Propagation 


Once  the  constrained  optimization  prodnces  a  resnlt,  we  nse  the  error  propagation  ap¬ 
proach  [9]  [14]  to  transform  the  inpnt  error  covariance  matrix  to  the  ontpnt  covariance 
matrix.  In  the  bnilding  estimation  problem,  we  have  the  optimization  model 

mm  /(©) 
snbject  to  h(0)  =  0 

where  /  is  the  snm  of  sqnared  errors  between  the  estimated  3D  points  and  the  observed 
3D  points. 

The  Lagrangian  fnnction  is 

T(X',0,A)=/(X',0)  +  A^h(0) 

Snppose  (X,  0,  A)  is  an  optimal  point.  From  the  necessary  conditions  on  a  local  minimnm 
point,  the  linearized  model  at  the  optimal  point  can  be  obtained  [13,  6]  by  solving 

Q  W  A0  \  /  -BAX  A 

H  o;vAAj“l^  0  ;■ 

The  Lagrangian  matrix  at  the  point  of  (X,  0,  A)  can  be  approximated  by  the  Lagrangian 
matrix  at  the  minimnm  if  the  error  is  small.  Hence  the  linear  model  can  be  approximated 

by 

/  Q*  (H*f  W  A0  \  /  -B*AX  \ 

H*  0  j  AA  j  0  J 

where 

Q*  =  V2,£(X',0,A) 

=  vV(x',0)  +  Ea,v2^,(0) 

i=i 

B*  =  V,2,£(X',0,A)  =  VL/(X',0) 

H*  =  Vh(0) 

Assnme  that  the  constraints  are  linearly  independent.  Then  the  row  vectors  in  matrix 
H*  are  linearly  independent.  We  can  nse  the  nnll  space  method  to  compnte  the  error 
propagation  matrix  J  [6]  [13]. 

0nce  the  error  propagation  matrix  is  obtained,  we  can  propagate  the  covariance 
matrix  of  the  observations  S  to  the  ontpnt.  The  covariance  matrix  of  the  estimated 
parameters  S©  can  be  approximated  by 

s©  =  .mr  (9) 
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8  Experimental  Methodology 

To  validate  the  optimization  algorithm  and  the  error  propagation  model,  an  experiment 
is  needed.  This  section  describes  the  experimental  methodology  for  this  validation. 

8.1  Ideal  Data  Generators  and  Noise  Model 

Three  building  types,  the  cube  box,  the  peak  roof  house,  and  the  hip  roof  house,  appear 
frequently  in  the  given  sites.  The  peak  roof  house  model  assumes  a  slanted  roof  on  a  cube. 
The  hip  roof  house  model  assumes  a  roof  with  slanted  sides  and  ends.  For  a  more  detailed 
description  of  the  models  and  corresponding  constraints  see  [13].  They  are  chosen  as  our 
prototype  models  with  unknown  length,  location  and  orientation  parameters. 

In  our  experiment,  ideal  data  generators  randomly  generate  the  ideal  parameters  for 
the  prototype  models  and  produce  ideal  3D  points. 

Assume  that  a  3D  coordinate  system  x-y-z  is  used.  To  simulate  the  site  model 
situation,  the  ground  is  assigned  as  the  plane  z  =  0.  Without  losing  generality  we 
assume  that  the  ideal  model  parameters  that  determine  the  3D  positions  of  the  building 
vertices  have  uniform  distributions.  The  center  of  the  bottom  plane  of  a  basic  model  is 
in  a  region  dehned  by  [— Xq,  Xq),  [— ?/o,  ?/o),  ^o)-  The  basic  model  is  rotated  on  the 

ground  by  a  random  angle  (j)  G  ?^i)- 

The  three  length  parameters  for  the  cube  box  model  are  denoted  by  a,  &,  c,  with 
Uo  <  a  <  Ui,  &o  A  b  <  &i,  and  Cq  <  c  <  Ci.  In  our  experiment  the  ranges  of  the 
parameters  for  this  model  were  set  as  follows: 


Xo 

yo 

zo 

<f>0 

h 

Go 

Ui 

bo 

bi 

Co 

Cl 

50 

50 

0 

0 

27r 

30 

60 

30 

60 

30 

60 

The  peak  roof  model  uses  four  length  parameters,  a,  &,  c,  d,  where  a,  &,  c  are  the  same 
as  those  in  cube  model  and  their  ranges  are  the  same.  The  height  of  the  peak  roof  is 
dehned  by  a  parameter  d,  with  fO  <  d  <  20.  The  hip  roof  model  requires  one  more 
parameter  e;  the  length  of  the  roof  edge  is  a  —  2e,  with  5  <  e  <  fO. 

For  each  building  type,  K  ideal  buildings  are  randomly  generated.  Fach  of  these 
K  buildings  will  be  used  in  n  experiments  in  which  Gaussian  random  noise  is  added 
to  each  of  the  3D  coordinates  of  the  building  and  constrained  optimization  is  used  to 
estimate  the  building  vertices  that  satisfy  the  geometric  constraints.  As  a  result  of 
these  n  experiments,  n  estimates  of  the  building  parameters  are  produced.  It  is  these  n 
estimates  on  which  the  hypothesis  test  statistics  will  be  computed.  We  call  the  procedure 
for  determining  these  n  test  statistics  a  trial.  Since  there  are  K  ideal  buildings  for  each 
ideal  building  type,  we  can  compute  K  test  statistics.  These  K  statistics  can  then  be 
used  to  test  the  hypothesis  that  their  distribution  is  what  the  statistical  theory  of  the 
test  says  it  should  be. 

The  noise  values  are  independently  sampled  from  a  Gaussian  distribution  A/"(0,  u^/), 
where  a  is  the  standard  deviation  of  the  random  variables  Sx^Sy^Sz.  We  repeated  each 
experiment  with  a  set  to  1.0,  2.0  or  3.0.  The  validation  results  for  all  three  different 
standard  deviations  are  similar,  so  here  we  discuss  only  the  validation  for  the  case  where 
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the  standard  deviation  is  eqnal  to  3.0.  K  is  set  at  100.  n  is  set  at  500  for  cnbe  model 
and  700  for  other  models. 

8.2  Statistical  Test 

In  each  trial  a  sample  of  model  parameters  and  corresponding  ideal  3D  points  s  prodnced 
by  the  ideal  data  generator.  Let  the  ideal  parameters  be  denoted  by  0.  For  each  ideal 
bnilding  instance  having  parameters  0,  n  independent  pertnrbations  {AX*,  i  =  1-,  ...,n} 
are  generated  from  the  noise  model  with  distribntion  A/"(0,  S).  By  adding  the  pertnr¬ 
bations  to  the  ideal  points,  the  pertnrbed  data  set  {X{,  X2, ...,  is  generated.  For 
each  of  the  pertnrbed  data  sets  {X{,X2,  ...,X^},  an  optimal  solntion  0j  is  compnted  by 
solving 


min  f{X',ei)  ^  =  l,...,n 
snbject  to  h(0i)  =  0 

Thns  we  have  n  estimates  {0^,  ii  =  1, ...,  n}. 

Using  eqnation  (9),  we  can  transform  the  inpnt  covariance  matrix  throngh  the  error 
propagation  matrix  to  the  ontpnt.  If  the  linear  model  is  valid,  the  estimated  parameters 
{0,  i  =  1,  shonld  be  approximately  distribnted  as  X"(0, 

Let  AQi  denote  0^  —  0,  i  =  1, ....,  n.  Let  //q  =  0  and  So  =  -/S./^.  Under  the  linearized 
model,  {A0q  i  =  l,...,n}  have  distribntion  So).  Considering  {A0i,  i  =  l,...,n} 

as  a  random  sample  from  a  Ganssian  distribntion  S),  we  can  perform  any  one  of 

the  hve  hypothesis  tests.  Here  we  jnst  discnss  onr  resnlts  for  hypothesis  H^\  fj,  =  fj,o  and 
S  =  So.  The  resnlts  for  the  other  hypothesis  tests  are  similar. 

The  signihcance  level  a  is  selected  to  be  0.05.  Under  the  nnll  hypothesis,  the  compnted 
statistics  of  the  mean  and  covariance  tests  have  the  nnll  distribntions.  This  can  be  verihed 
by  nsing  a  Kolmogorov- Smirnov  test  (K-S  test)  on  the  K  test  statistics  generated  from 
the  K  trials. 


8.3  Range  Space  Analysis 

The  standard  hypothesis  test  methods  reqnire  that  the  covariance  matrix  be  positive 
dehnite.  However,  becanse  of  the  constraints,  the  ontpnt  covariance  of  a  constrained 
optimization  is  generally  positive  semi-dehnite. 

Theorem  1  Suppose  that  not  all  of  the  derivatives  of  the  constraint  equations  are  equal 
to  zero  at  the  local  minimum  point,  then  the  propagated  error  covariance  JYjJ^  is  singular. 


Proof  :  From  the  given  condition  we  know  that  the  derivative  matrix  H  is  not  a  zero 
matrix,  i.e.. 


/  An  \ 

/  90  \ 


H  = 


7^0 


\  An  / 

\  90  ^ 


fO 


Left  multiply  equation  A0  =  JAX  by  H  : 


HAQ  =  HJAX 

Because  both  (X,  0*,  A*)  and  [X  +  AX,  0*  +  A,  A*  +  AA)  are  local  minimum  points  of 
the  optimization,  the  following  equation  is  satished, 

HAQ  =  0. 


Hence 


0  =  HJAX 


Since  the  formula  holds  for  any  AX,  this  implies  that 


0  =  HJ  (to) 

We  now  use  this  result  to  prove  that  JYjJ^  is  singular.  Left  multiply  JYjJ^  by  H  and 
right  multiply  it  by  .  From  (tO)  we  have 

HJ^J^H^  =  OSO 


Since  H  is  not  a  zero  matrix,  JYjJ^  must  be  singular.  q 

To  utilize  standard  hypothesis  technology,  we  project  a  positive  semi-dehnite  matrix 
onto  its  range  space.  Suppose  that  an  n  X  n  covariance  matrix  So  has  k  nonzero  eigen¬ 
values  rci, ...,  rci;  and  associated  unit  eigenvectors  Ui, ...,  A  basis  of  the  range  space  of 
So  can  be  dehned  by 

B  =  (ui,  ...,n„) 

Use  B  to  perform  a  matrix  transformation  as  follows: 


H^SoH  =  Sb 


!  wi  Q 
I  0  ?n2  0 


\  0  .  .  .  Wk  j 


Let  B^  be  a  basis  matrix  of  the  null  space  of  matrix  So.  It  is  obvious  that  [B^B^) 
is  orthonormal.  In  our  experiment,  we  check  whether  [B^Y AQ  has  very  small  vari¬ 
ances  (caused  by  round-otf  errors  and  nonlinear  terms).  If  this  is  true,  we  conduct  the 
hypothesis  test  on  variables  B^ AQ  with  covariance  matrix  S^. 

Due  to  the  round-otf  errors  and  nonlinear  terms,  the  zero  eigenvalues  of  matrix  So 
may  not  be  exactly  zero.  We  use  a  small  threshold  to  distinguish  the  zero  eigenvalues 
from  the  nonzero  eigenvalues.  In  all  our  experiments  the  threshold  was  set  to  10“®  times 
the  maximum  eigenvalue. 

For  the  cube  model,  the  range  space  of  the  output  error  covariance  matrix  has  7 
dimensions.  This  can  be  understood  as  follows.  Consider  a  cube  house  whose  faces  are 
all  at  right  angles  to  each  other.  Count  the  number  of  degrees  of  freedom.  The  size  of 
a  cube  model  is  dehned  by  three  independent  parameters.  The  location  of  the  model  is 
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specified  by  three  translation  parameters  in  3D  space.  In  onr  experiments,  the  normal 
vector  of  the  cnbe  roof  is  hxed  in  the  vertical  direction.  The  only  possible  rotation  is 
aronnd  the  vertical  axis  of  the  model.  Thns  the  total  nnmber  of  independent  parameters 
is  seven.  For  the  peak  roof  model  the  analysis  is  similar,  except  that  two  more  parameters 
are  needed  to  determine  the  roof  height  and  the  ridge  position.  (In  the  partial  model 
we  do  not  hx  the  horizontal  position  of  the  roof  ridge  to  the  center  of  the  bnilding.) 
Thns  the  range  space  of  the  ontpnt  covariance  matrix  for  the  peak  roof  model  has  nine 
dimensions.  The  hip  roof  model  inherits  all  the  parameters  of  the  peak  roof  model.  It 
reqnires  two  more  parameters  to  determine  how  mnch  of  the  ridge  was  cnt  of  at  each  of 
the  two  ends  (they  are  assnmed  to  be  independent).  These  parameters  can  be  thonght 
of  as  the  relative  starting  and  ending  points  of  the  the  ridge.  Thns  the  range  space  of 
the  ontpnt  covariance  matrix  for  the  hip  roof  model  has  11  dimensions. 

9  Experimental  Results 

In  this  section  we  present  onr  experimental  resnlts  on  the  cnbe  model,  the  peak  roof 
model  and  the  hip  roof  model.  An  example  of  the  model  estimation  resnlts  is  shown  in 
Fignre  4. 

9.1  Test  of  Cube  Model  with  a  =  3.0 

The  theoretical  and  empirical  nnll  distribntions  of  the  hve  test  statistics  for  the  cnbe 
model  are  shown  in  Fignre  1.  The  standard  deviation  nsed  was  a  =  3.0.  The  x  axis  is 
the  statistic  nsed  in  the  test  and  the  y  axis  represents  1  —  a,  where  a  is  the  signihcance 

level. 

The  experimental  trials  were  rnn  mnltiple  times;  each  time,  the  nnll  hypothesis  was 
either  rejected  or  not  rejected.  The  resnlts  are  summarized  in  Table  1(a).  The  nnll 
hypothesis  is  not  rejected  at  a  0.05  signihcance  level. 

The  K-S  test  was  nsed  to  test  whether  the  empirical  and  theoretical  distribntions  are 
similar.  For  this  test  the  nnmber  of  degrees  of  freedom,  p,  is  7.  The  resnlts  are  shown  in 
Table  2(b).  None  of  the  hve  test  statistic  distribntions  failed  the  K-S  test  at  a  signihcance 
level  of  0.05.  Thns  the  optimization  model  and  the  error  propagation  model  for  the  cnbe 
model  were  validated. 

9.2  Test  of  Peak  Roof  Model  with  a  =  3.0 

The  theoretical  and  empirical  nnll  distribntion  of  the  hve  test  statistics  for  the  peak  roof 
model  are  shown  in  Fignre  2.  The  standard  deviation  nsed  was  a  =  3.0.  The  x  axis  is 
the  statistic  nsed  in  the  test  and  the  y  axis  represents  1  —  a,  where  a  is  the  signihcance 

level. 

The  experimental  trials  were  rnn  mnltiple  times;  each  time,  the  nnll  hypothesis  was 
either  rejected  or  not  rejected.  The  resnlts  are  summarized  in  Table  2(a).  The  nnll 
hypothesis  is  not  rejected  at  a  0.05  signihcance  level. 

The  K-S  test  was  nsed  to  test  whether  the  empirical  and  theoretical  nnll  distribntions 
are  similar.  For  this  test  the  nnmber  of  degrees  of  freedom,  p,  is  9.  The  resnlts  are 
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Figure  1:  Null  distributions  of  the  five  test  statistics  for  the  cube  model  at  cr  =  3.  (a) 
Test  mean  with  known  covariance,  (b)  Test  mean  with  unknown  covariance,  (c)  Test 
covariance  with  known  mean,  (d)  Test  covariance  with  unknown  mean,  (e)  Test  mean 
and  covariance. 
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Table  1:  Cube  model. 


(a)  Test  Statistic  Values  for  the  Five  Tests 


test  method 

number  of  trials 

sample  size 

signihcance  level 

reject  rate 

test  //,  known  S 

100 

500 

0.05 

0.06 

test  //,  unknown  S 

100 

500 

0.05 

0.05 

test  S,  known  // 

100 

500 

0.05 

0.07 

test  S,  unknown  // 

100 

500 

0.05 

0.06 

test  fi  and  S 

100 

500 

0.05 

0.05 

(b)  Kolmogorov- Smirnov  Distribution  Test 


test  method 

null  distribution 

size  n 

p- value 

K-S  test 

test  //,  known  S 

xi 

100 

0.391 

pass 

test  //,  unk.  S 

p,n—p 

100 

0.217 

pass 

test  S,  known  // 

Xl{p+l)/2 

100 

0.866 

pass 

test  S,  unk.  // 

Xp(p+l)/2 

100 

0.881 

pass 

test  fi  and  S 

Xp(p+l)/2+p 

100 

0.700 

pass 

shown  in  Table  2(b).  None  of  the  hve  test  statistic  distributions  failed  the  K-S  test  at  a 
signihcance  level  of  0.05.  Thus  the  optimization  model  and  the  error  propagation  model 
for  the  peak  roof  model  were  validated. 

9.3  Test  of  Hip  Roof  Model  with  a  =  3.0 

The  theoretical  and  empirical  null  distributions  of  the  hve  test  statistics  for  the  hip  roof 
model  are  shown  in  Figure  3.  The  standard  deviation  used  was  a  =  3.0.  The  x  axis  is 
the  statistic  used  in  the  test  and  the  y  axis  represents  1  —  a,  where  a  is  the  signihcance 

level. 

The  experimental  trials  were  run  multiple  times;  each  time,  the  null  hypothesis  was 
either  rejected  or  not  rejected.  The  results  are  summarized  in  Table  3(a).  The  null 
hypothesis  was  not  rejected  at  a  0.05  signihcance  level. 

The  K-S  test  was  used  to  test  whether  the  empirical  and  theoretical  null  distributions 
are  similar.  For  this  test  the  number  of  degrees  of  freedom,  p,  is  ff.  The  results  are 
shown  in  Table  3(b).  None  of  the  hve  test  statistic  distributions  failed  the  K-S  test  at  a 
signihcance  level  of  0.05.  Thus  the  optimization  model  and  the  error  propagation  model 
for  the  hip  roof  model  were  validated. 

10  Summary 

We  have  described  a  statistical  methodology  for  validating  the  theoretical  derivations  and 
software  that  make  up  a  system  for  estimating  3D  positions  of  building  vertices,  based 
on  input  data  obtained  from  multi-image  photogrammetric  resection  calculations.  Frror 
propagation  allowed  us  to  derive  the  null  distributions  of  various  test  statistics  of  mea- 


f4 


1.0 

2.5 

' 

' 

sample  density 

0.8 

- 

2.0 

- sample  distribution 

- null  distribution  function 

- null  density  function 

0.6 

j sample  density 

1.5 

//  -  sample  distribution 

.c 

j!  - null  distribution  function 

/'  - null  density  function 

0.4 

/ 

1.0 

- 

0.2 

0.5 

0.0 

0.0 

0.0  10.0  20.0  30.0  ■  0.0  1.0  2.0  3.0  4.0 


statistic 


statistic 


1.0 

(a) 

1.0 

(b) 

0.8 

0.8 

/ 

/ 

/ 

0.6 

sample  density 

0.6 

'7  sample  density 

.c 

- sample  distribution 

.c 

1;  - sample  distribution 

- null  distribution  function 

% 

:  - null  distribution  function 

/  - null  density  function 

//  - null  density  function 

0.4 

0.4 

! 

0.2 

0.2 

/ 

a' 

0.0 

0.0 

20.0  40.0  60.0  80.0  '  20.0  40.0  60.0  80.0 

statistic  statistic 


(c) 


(d) 


sample  density 

-  sample  distribution 

-  null  distribution  function 

-  null  density  function 


20.0  40.0 


60.0 

statistic 


80.0  100.0 


(e) 


Figure  2:  Null  distributions  of  the  five  test  statistics  for  the  peak  roof  model  at  cr  =  3. 
(a)  Test  mean  with  known  covariance,  (b)  Test  mean  with  unknown  covariance,  (c)  Test 
covariance  with  known  mean,  (d)  Test  covariance  with  unknown  mean,  (e)  Test  mean 
and  covariance. 
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Figure  3:  Null  distributions  of  the  five  test  statistics  for  the  hip  roof  model  at  cr  =  3.  (a) 
Test  mean  with  known  covariance,  (b)  Test  mean  with  unknown  covariance,  (c)  Test 
covariance  with  known  mean,  (d)  Test  covariance  with  unknown  mean,  (e)  Test  mean 
and  covariance. 
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Figure  4:  This  figure  shows  the  estimated  houses  overlaid  on  top  of  the  original  model- 
board  image. 
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Table  2:  Peak  roof  model  results. 


(a)  Test  Statistic  Values  for  the  Five  Tests 


test  method 

number  of  trials 

sample  size 

signihcance  level 

reject  rate 

test  //,  known  S 

too 

700 

0.05 

0.03 

test  //,  unk.  S 

too 

700 

0.05 

0.03 

test  S,  known  // 

too 

700 

0.05 

0.04 

test  S,  unk.  // 

too 

700 

0.05 

0.04 

test  and  S 

too 

700 

0.05 

0.02 

(b)  Kolmogorov- Smirnov  Distribution  Test 


test  method 

null  distribution 

size  n 

p-value 

K-S  test 

test  //,  known  S 

xi 

too 

0.464509 

pass 

test  //,  unknown  S 

F 

p,n—p 

too 

0.277997 

pass 

test  S,  known  // 

v2 

Xp{p+l)/2 

too 

0.450701 

pass 

test  S,  unknown  // 

Xp{p+l)/2 

too 

0.402053 

pass 

test  fi  and  S 

Xp(p+l)/2+p 

too 

0.205762 

pass 

surable  quantities.  These  theoretically  derived  null  distributions  allowed  us  to  validate 
whether  the  measurements  in  the  system  followed  the  theoretically  derived  distributions. 
Kolmogorov- Smirnov  tests  were  performed  to  check  whether  the  empirical  and  theoreti¬ 
cal  distributions  were  close  enough.  None  of  the  empirically  computed  null  distributions 
failed  the  KS  test.  Thus  we  have  conhrmed  that  the  theoretical  derivations  of  the  null 
distributions  were  correct  and  that  the  software  implementing  the  theory  is  also  correct. 
The  software  for  performing  the  statistical  tests  has  been  made  publicly  available. 
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